The present invention relates generally to parallel programming techniques, and more particularly, but not by way of limitation, to a system and method for programming a cluster similarly to a large shared memory system which employs a parallel programming model that does not require coherence.
Conventionally, parallel programming paradigms are partitioned into two classes, shared memory and message passing. In shared memory paradigms, the individual execution sequences, usually called “threads”, semantically all operate on the same ‘memory’. In message passing paradigms, the individual execution sequences, usually called “processes”, operate on separate ‘memories’. The two classes of paradigms have different advantages and disadvantages.
Maintaining coherence is a challenge for shared memory systems with many processors, and even more so in a cluster of distributed systems. The message passing paradigm does not require coherence, but at the price of higher cost of data exchange. Data structures have to be serialized and de-serialized in order to be exchanged between processes. Furthermore, a data exchange between processes in different compute nodes requires a copy.
Synchronization errors as, for example, race conditions and deadlocks are often hard to find in shared memory applications. Message passing paradigms avoid race conditions, but data structures have to be partitioned such that each part fits into the memory of a single compute node.